Vehicle remote guidance with path control

ABSTRACT

In a remotely piloted surface vehicle a television camera is used to send images to a screen on the operator&#39;s station. A computer displays the vehicle&#39;s intended path on the screen. The path appears as a computer-generated line superimposed on the image of the vehicle&#39;s environment, like a stripe painted on the ground. The operator can change or advance the path on the screen with a cursor control. A computer picks certain discrete screen points along the line and maps those screen positions onto ground positions called &#34;waypoints&#34;. These are sent to the vehicle&#39;s guidance system to direct the vehicle along a path through the waypoints. The transform which maps the screen path onto the ground path depends on the camera orientation and lens. The transform parameters can be adjusted as the camera zooms, pans and tilts. Each time the screen is refreshed, the path line for that screen is calculated by the computer from the ground path, to present the ground path correctly in the new screen. The operator can extend or modify the path at will. The system is especially adapted to use in cases where a narrow bandwidth of the radio link between the camera and the station limits the screen refresh rate. The system maximizes the possible speed of the vehicle by presenting the path information in a format quickly grasped by the operator.

FIELD OF THE INVENTION

The present invention relates to remote control systems for unmannedsurface vehicles which are guided by an operator, in systems where theoperator can see the vehicle environment by a television link.

DESCRIPTION OF THE PRIOR ART

Remote vehicle guidance has many areas of possible application. Anoperator riding within the vehicle may be impractical if the vehicle issmall or underpowered. A vehicle operator may also be endangered inpoisonous environments or in military vehicles during a battle.

A system for remote control must transmit commands from the operator'sstation to the vehicle, and the operator also needs feedback from thevehicle. The typical way to feed back information to the operator isthrough a video link from a television camera on the vehicle. Televisioncameras are now small and rugged enough for almost any application. Itmight be expected that an operator sitting at a remote station couldcontrol a remote vehicle as well by watching the environment through ahigh-resolution TV screen as by sitting in the vehicle and lookingthrough a window. Unfortunately, however, the usefulness of remotelycontrolled vehicles is often limited by poor image feedback to theoperator. This is because radio links are the only practical way to sendback the images from the TV camera on the vehicle to the operator'sstation, in most cases.

Television, or any other system which transmits images, requires a largeradio bandwidth because information is transmitted at a high rate.Images which are sent in real time, and which have reasonableresolution, may require a megahertz or more of bandwidth. Clearly,finding a place in the radio spectrum for the video link can be hard. Inthe case of military vehicles, there could be hundreds or even thousandsof vehicles all operating at once during a battle, each needing its ownuncluttered band.

Single-sideband radio, data compression techniques, and computer imageenhancement can help somewhat. Special techniques like fiber-optictethers and laser or microwave links could entirely solve the bandwidthproblem, but they are impractical for other reasons. Tethers areobviously limited by length, fragility, and fouling. Infrared, visibleand microwave links can only be used as line-of-sight beams, which mustbe accurately aimed at the moving vehicle and which are cut off byintervening objects, including the horizon. For radio links of ordinaryfrequencies and ranges, the fundamental constraints imposed by themathematics of Fourier's Theorem and information theory often will meanthat the images coming to the operator are either very grainy, oralternatively, that they can only be refreshed at intervals greater thanthe persistence time of the eye (about a twentieth of a second) and sowill flicker or appear as a series of still images. Image resolution andrefresh rate can be traded off against one another, but the graininessof the image cannot be arbitrarily increased; so with limited bandwidth,the refresh rate will be slowed.

(Even if the refresh rate is of the order of a minute, usefulinformation will be presented at intervals; but if the grain is toocoarse, objects will not be seen and the operator will end up collidingthe vehicle with some obstacle.)

Once the screen refresh rate drops to the point where a second or moreis elapsing between frames, driving the remote vehicle becomes verydifficult. The operator will tend to react to a screen as if it is inreal time, that is, as if the vehicle were at the ground position fromwhich the screen image is taken; but the vehicle is elsewhere by thetime the operator views the image. A delay is introduced into thefeedback loop which sends the operator's commands to the vehicle and thecamera's information to the operator. If the operator is under stress ormust quickly make many decisions, as in a battle, he or she is likely tocontrol the vehicle badly--even if trained to take the delay intoaccount. As the delay becomes longer, the problem is aggravated.

The prior art has dealt with this problem in various ways. One approachis to limit high resolution to only a portion of the viewing screen;this portion is picked by the operator to view an area of interest. Thearea of interest appears in focus, surrounded by blurred area. Thisapproach is discussed by Kanaly in U.S. Pat. No. 4,405,943. The loss ofinformation with this system is obvious. Complication and operatorconfusion are introduced by the requirement of picking an area, and theextra hardware and/or software required.

Graham, in U.S. Pat. No. 4,682,225, discloses an other system. Grahamdiscusses image compression, which involves sampling the data streamfrom the camera at intervals and transmitting only the sampled data. Theresulting screen image is blurred. If the camera is almost stationary,image clarity can be improved by superimposing the blurred images of thesampled data. (The human eye will itself do this over intervals lessthan a second.) Basically, this system trades off clarity or detail infavor of an appearance of continuous "real time" motion on the screen.The same bandwidth which could transmit high resolution screens atintervals instead transmits a multitude of blurred screens. If thecamera is panned, zoomed, or jiggled, the technique is totallyineffective. Also, if the superposition is by hardware or softwarerather than in the eye, cost and complexity are involved.

Hinman, in U.S. Pat. No. 4,661,849, discusses interpolation betweendiscrete screen images or frames by computer simulation. This presentsan appearance of smooth motion to the viewer. Such a system is costly incomputer hardware and software running time, and may mislead theoperator by presenting an impression of real time events which arefictitious projections instead of the real environment.

Narendra et al. (U.S. Pat. No. 4,855,822) also employs a computer togenerate interpolated image between the discrete images sent by thecamera, so as to present an impression of continuous motion of theoperator. Their interpolations are determined by the motion of thevehicle. Narendra et al. also disclose the idea of superimposing animage of the vehicle on the screen. Conventional bandwidth compressiontechniques are used by Narendra et al.

The Jet Propulsion Laboratory has developed a system called ComputerAided Remote Driving (CARD). The CARD system is described in a paper,"Computer-Aided Remote Driving", presented at the 13th annual meeting ofthe Association for Unmanned Vehicle Systems in Boston, MA on Jul.21-23, 1986. The paper is authored by Brian H. Wilcox, Robert Salo,Brian Cooper, and Richard Killon, Technical Group Supervisor at the JetPropulsion Laboratory of California Institute of Technology in Pasadena,CA.

CARD is intended for interplanetary remote control of vehicles andmilitary applications. In remotely driving a vehicle on another planet,narrow bandwidths data restrictions are compounded by message delays dueto the finite speed of radio beams. Vehicle speed is not crucial onanother planet, but may be in a military application.

The CARD system uses two high-resolution cameras to generate a stereoimage for the operator. The operator views both images at once, onethrough either eye, to see the vehicle environment in three dimensions.The viewing system has two screens with relatively crossed Polaroidfilters, two half-silvered mirrors to superimpose the images, andPolaroid glasses worn by the operator to isolate the two images.Three-dimensional viewing may be helpful when the operator is viewing anextraterrestrial environment, and is less able to extrapolate distancesfrom unfamiliar objects.

The CARD operator at the control station sends a signal to the vehicleto transmit the stereo images, and waits for all the data for bothscreens to arrive at the station and to appear in the stereo viewer.Then the operator uses a three-dimensional control to denote points inthe space seen in the viewer. (A three-dimensional control is one withthree degrees of freedom; CARD uses a joystick with a rotatable knob.)The control drives a cursor which is superimposed on the picture whichthe operator sees, and which appears to move about in space in responseto the operator's motion of the three-dimensional control.

A computer at the station takes the three dimensions of joystick motionand turns them into Cartesian space coordinates x, y, z at the vehiclelocation; it then transforms those coordinates into individual screenpositions for the two viewing screens, so that the operator sees thecursor located in space in the stereo image. The transform from spacecoordinates to screen coordinates can easily be programmed from thegeometry.

The operator, by depressing a button, can denote any cursor position asa waypoint. He or she denotes a series of waypoints to define points ofa path in the space seen in the viewer, over which the operator wantsthe vehicle to travel. When all the waypoints are denoted, the operatorpushes the "go" button. The station computer then takes the controlreadings recorded from the waypoints, transforms them into theappropriate commands (vehicle angles, segment lengths, compassheadings), and relays these commands to the vehicle. The receivedcommands tell the vehicle's guidance system how to proceed. The vehicleautomatically responds to the commands by moving to the next waypoint;eventually it reaches the final point. It then begins the process overby sending two more images.

Neither the station computer nor the on-board computer calculates anycurve from the waypoints: the vehicle moves straight to the next point,turns abruptly, and then goes on to the next. The station computerinterrogates the vehicle's on-board computer about the vehicle's headingafter each leg of the path is traversed, and instructs the vehicle forthe next leg of the path.

CARD avoids the feedback problem by eliminating any semblance ofreal-time driving, and instead presenting the operator with a staticproblem: given a picture, chart a path through it. Operator confusion iseliminated, but at the cost of dead-time in the control cycle. Theoperator must wait while the vehicle laboriously goes through all of thewaypoints, takes a picture, and transmits the image over the slow radio.

Being sluggish, CARD is not adapted to any use in which the operatorshould react quickly to changes in the vehicle environment, such asmilitary use. CARD also has the drawback that it effectively halves thebandwidth of the radio link by presenting two stereo images, instead ofonly one. Moreover, the resolution needed for each of the combinedstereo images is substantially greater than the resolution needed for asingle monoscopic image of equal clarity. This is because higherresolution is needed to locate objects in the depth dimension when theimages are combined. This need further decreases the effectivebandwidth.

CARD uses conventional data compression techniques to decrease thebandwidth by about a factor of four. Such techniques are too slow forreal time video, but are effective with slower transmission.

The CARD prototype described in the paper uses solid-state cameras 0.5 mapart. The cameras have a grain of 320 pixels per horizontal line,giving a 1-pixel stereo offset for objects at a range of 300 m. Thevehicle includes a magnetic compass and odometer for dead reckoningcalculation of vehicle position by the small on-board computer.

None of the above inventions and patents, taken either singly or incombination, is seen to describe the instant invention as claimed.

The prior art does not disclose any system for driving a vehicle byremote imaging under low screen refresh rates which is adapted toreal-time driving; which is easy and natural for an operator to use;which is simple and reliable; which is inexpensive; and which allows theoperator to react in the least possible time.

Accordingly, one object of the present invention is a vehicle remoteimaging control system which does not confuse the operator with timedelays in the control loop.

Another object is a system which is as simple as possible given theconstraints of slow image data transfer, so as to be reliable andinexpensive.

A further object is a system with minimal computer hardware and softwarerequirements.

An additional object is a system which allows a vehicle to proceed atthe utmost speed.

A final object is a system which uses only available, proven technology.

These and other objects of the present invention will become readilyapparent upon further review of the following specification anddrawings.

SUMMARY OF THE INVENTION

In a remotely piloted surface vehicle, where a television camera on thevehicle is used to send images to a screen at a vehicle operator'sstation, the present invention comprises an improved and simplifiedsystem for remote driving. The system is especially adapted to slowvideo data transfer rate situations, where real-time video isunavailable and the operator can see only discrete "snapshot" imageframes on the screen.

The vehicle's intended path is displayed on the operator's viewingscreen. The path appears as a computer-generated line superimposed onthe image of the vehicle's environment, appearing like a stripe paintedon the ground. A screen cursor appears at the end of the line. Theoperator can change or advance the path line on the screen with a cursorcontrol device, which might be a joystick, mouse, steering wheel andpedals, or any other control having two degrees of freedom.

As the line is extended by the operator, the computer picks certaindiscrete screen points along the line extension. The computer then mapsthese points onto ground positions in the vehicle environment by amathematical transform. The ground positions are called "waypoints".These are sent to the vehicle's guidance system to direct the vehiclealong a path through the waypoints. The guidance system has a memorywhich stores the waypoints and directs the vehicle successively overthem.

The transform which maps the screen path onto the ground path usessimple trigonometric formulas and perhaps coordinate transformations.The transform and parameters depend on the camera orientation and lens.The transform parameters can be continuously adjusted if the camerazooms, pans or tilts.

In the usual low video data rate situation, the operator will see asequence of still frames or "snapshots". A few frame will replace theold automatically at intervals determined by the data rate and thescreen resolution.

The vehicle's computer includes an image buffer to store data from ainstantaneous view of the camera. This data is sent to the station. Onceall the data are arrived, that frame is automatically displayed on theoperator's screen. The operator sees a frame taken some time ago.

For each new screen the path line is recalculated from the reportedposition of the vehicle relative to the ground points. The recalculatedpath line is then superimposed on the screen so as again to appear tolie on the surface, and the operator can quickly perceive the newsituation of the vehicle and correct the projected path.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an operator controlling the radiomessage from a computer.

FIG. 2 a perspective view of the controlled vehicle, showing obstacles,a trapezoidal area seen by the camera on the vehicle, and waypoints onthe ground.

FIG. 3 is a schematic elevation view showing a screen path andrectangular coordinates on the screen.

FIG. 4 is a schematic plan view showing the rectangular coordinates andscreen path of FIG. 3 transformed into ground coordinates and waypointson the ground.

FIG. 5 is a perspective view of the vehicle.

FIG. 6 is a schematic showing the flow of vehicle control informationthrough the system.

FIG. 7 is a schematic showing the flow of camera control informationthrough the system where the camera is movable relative to the vehiclebody.

Similar reference characters denote corresponding features consistentlythroughout the attached drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Some definitions of terms, used in the following Description and Claims,are:

"Camera" means film camera, infrared TV, radar, imaging sonar, or anyother imaging apparatus whatsoever, as well as the typical visible lightscanning television camera.

"Computer" means a digital computer, analog computer, hard-wired logiccircuits, or any calculating device with memory.

"Cursor" means any visible, movable highlight, emblem, outline shape,etc. which can denote a position or area on a screen, and which can movein at least two dimensions. The cursor may include means for denotingdirection as well as position, particularly by its shape. Typically, thecursor will be a flashing highlight on a CRT screen. The cursor may beshaped as a point, an arrow, an outline of the vehicle as seen inperspective at the cursor location, or any other shape.

"Downlink" refers to radio transmission of information from the vehicleto the station.

"Frame" means a single static view or image which fills the screen.

"Ground" means a surface over which the vehicle moves, be it concrete,carpeting, water, or any other surface. The invention is not limited toland vehicles traveling over earth.

"Radio" means communication by waves, such as electromagnetic waves.Sonar is one possible such wave.

"Screen" means any viewing device which appears to be flat ortwo-dimensional to the operator or viewer (as opposed to stereoscopic).Ordinarily the screen of the present invention will physically be asurface, such as the glass surface of a CRT. A liquid crystal screen, orany other surface capable of displaying a path and some imagecorresponding to the vehicle environment, may be used. Emblems or iconswhose positions correspond to those of objects or conditions near thevehicle may be used instead of, or in addition to, regular transmittedpictures. The definition does not exclude monocular viewers such ashelmet-mounted devices, or viewing goggles with individual lens systemspresenting the same image, which do not incorporate a physical surfacefor viewing. Enhanced images are also within the scope of the invention.

"Uplink" refers to radio transmission of information from the station tothe vehicle.

"Vehicle" means any device adapted to travel on or above a groundsurface, having means to power the device to move and having means forcontrolling the motions.

"Waypoint" means a ground position referable to the coordinates of thevehicle guidance system or steering geometry. A waypoint may also be aposition combined with a heading and/or a speed. It may also be any timederivative of position, such as acceleration. In general, it is a datumor data set to which vehicle position is referable. A waypoint mayinclude elevation as well as ground position.

The present invention, as seen in FIGS. 1 and 2, is a system forpresenting future vehicle path information to the operator O (FIG. 1) ofa remotely controlled vehicle V (FIG. 2), where the vehicle V sendsimages back to the operator's station S from a camera 30 mounted on thevehicle V. It employs a station computer 16 to superimpose on a stationscreen 14 a screen path line 12 showing the intended future path of thevehicle V over the surface. The operator O takes the line 12 to lie onthe ground surface, like a painted stripe on a highway; the system isdesigned for this operator perception, and includes software to generatea projected ground path as a line on screen.

Radios 24 at on the vehicle V and at the station S send data back andforth.

The operator O, seen at the control station in FIG. 1, uses a cursorcontrol joystick 10 to extend or modify the screen path line 12 as seenon the screen 14. The operator O traces out an apparent path for thevehicle V with the aid of a cursor 18 which is superimposed on theviewing image at the end of the line 12. The operator O can move thecursor 18 about at will with the joystick 10.

The line 12 must be transformed into a planned ground surface path by asimple computer program, which maps the position of any point on thescreen 14 into a corresponding ground point on the ground traversed bythe vehicle V. The mapping transform will ordinarily map the rectangleof the screen onto the trapezoid on the ground which is seen by thecamera 30, as shown in FIG. 2. The mapping program parameters willdepend upon the altitude, attitude, and focal length of the camera lens36.

The transform idea is illustrated schematically in FIGS. 3 and 4. FIG. 3shows the viewing screen of FIG. 1, but no camera image of the vehicleenvironment is shown: instead is shown a rectangular grid correspondingto a possible system of screen coordinates. (This grid is not a physicalimage seen by the operator O, but is rather a mathematical projection toexplain the transform.) A screen path is also shown, in the form of aline 12. The screen path ends with a cursor 18 in the shape of an arrow.FIG. 4 is a bird's eye or plan view of the ground in front of thevehicle V, showing the rectangular grid of FIG. 3 transformed into atrapezoidal grid. The screen path of FIG. 3 has been transformed intodiscrete "waypoints", labeled 20, which outline the planned ground pathof the vehicle; the generation of the waypoints 20 is explained below.Both the grid lines and waypoints of FIG. 4 are, like the grid of FIG.3, non-physical.

The transform is performed by a computer or computers at the station.This function is illustrated by the box labeled coordinate transform andwaypoint generator" in Schematic FIG. 6. FIG. 6 shows the flow ofinformation in the present invention: arrows represent information, andboxes are devices and/or processes such as computer algorithms. Thereader will find reference to FIG. 6 helpful in the followingdiscussion.

It is possible for the operator's cursor control to directly generateplanned ground coordinates. The computer would in this case transformthose coordinates back to screen coordinates for screen 14 display. Theessential thing is that the operator work in the view of the screen andthat the screen path be correctly transformed into a planned groundpath.

Usually the transform will require only trigonometry. If an unusualcamera lens 36 is used, for example a fisheye lens, more complexcoordinate transformations will be needed. In any case, the formulas oftransformation are simple, straightforward, and easy for one skilled inthe art to implement.

The control joystick 10 shown in FIG. 1 is well adapted to a simplepoint cursor 18. Other controls might be adapted to other sorts ofcursors. For example, if an angle control such as a steering wheel isused, the cursor 18 might be in the shape of an arrow, whose anglecorresponds to the angle of the steering wheel. The joystick 10 of FIG.1 can include a rotatable knob for angle.

The cursor 18 may also be made large and shaped like the outline of thevehicle V. This form of cursor 18 is illustrated in FIG. 1. The operatorO may then guide the cursor 18 outline through narrow gaps as shown inFIG. 1, and avoid a path if the outline will not fit through. In thiscase there could be two screen path lines 12 instead of the oneillustrated in FIGS. 1 and 3, each one line trailing from a respectiveouter edge of the vehicle outline cursor 18. The outline should bedisplayed in perspective, so as to appear on the screen image of theground as would a shadow of the vehicle V cast by a sun directlyoverhead. The size and shape of the outline can easily be calculated forthe display. Such an outline is easily programmed by one skilled in theart. The outline of the vehicle V may also be simplified to a rectangleor bar indicating the width of the vehicle V.

Preferably, the cursor 18 sweeps out the line 12 on the screen, just asa marker leaves a line on paper, to denote the screen path. As analternative, the cursor 18 might be used to set the screen positions ofthe individual ground waypoints 20 through which the vehicle V wouldpass, like dots on paper. The screen points could be clicked on with abutton and appear on the screen as dots or cursor shapes. The computer16 would then transform the coordinates of the screen path points intoground waypoint coordinates for radio uplink transmission to thevehicle's guidance system. However, operator choice of waypoints 20 putsan extra workload on the operator, so the waypoint choice is best leftto the station computer, which can pick waypoints 20 without inordinatehardware and software requirements.

If the path line 12 is continuous, the station computer 16 will pickground waypoints 20 based on some decision algorithm which keeps thewaypoints 20 spaced closely enough to track the vehicle V. The infinityof points defining a continous line cannot be transmitted to the vehicleV, and is not needed. The guidance system of the vehicle can regeneratea planned ground path through the waypoints 20 and then follow thatpath. The required spacing of the transmitted waypoints 20 will dependupon the sophistication of the path regeneration program of thevehicle's guidance system.

The coordinate transform and waypoint generation computer may displaythe screen path swept out by the cursor 18 as a line 12, or, maygenerate multiple images on the screen, located at the waypoint orelsewhere. Any display which informs the operator O of the screen pathof the vehicle is within the scope of the present invention.

Since it may be necessary to readjust the projected path of the vehiclein the face of emergencies or miscalculations, the cursor control shouldhave the capability of erasing the end of the screen path, that is,"back-tracking". It may be helpful to have a separate cursor reversecontrol which would erase the screen path line 12 from the end of theline back toward the vehicle.

The vehicle guidance system, which receives the waypoint coordinatesfrom the station's transform and waypoint generation computer by way ofthe radio uplink, must include a guidance system computer 26 as seen inFIG. 1, or equivalent hard-wired circuitry, which has a memory. Thememory stores the coordinates of the projected waypoints 20, whichdefine a ground path for the vehicle V. This allows the operator O toset a planned ground path for the vehicle V to follow while the nextframe is being transmitted. The guidance memory is an essential elementof the present invention, as it allows driving in the future time of theinstant of the snapshot frame, and avoids feedback loop trouble.

Because the waypoints 20 need to be erased when the path is changed, theguidance system computer memory may conveniently be of the last-in,first-out stack type.

The guidance system may of any type which allows the vehicle V toautomatically steer through the waypoints 20. The vehicle may rely ondead reckoning to guide it on its path along the waypoints. Any methodof tracking distance and direction, such odometers, integratingaccelerometers, compasses, gyroscopes, or other conventional guidancemeans are feasible.

The guidance system need not receive operator commands directed towardthe motion parameters of the vehicle V, such as vehicle speed andsteering wheels angle. The transmitted waypoints 20 are sufficient totrack the vehicle V and control its speed (although direct speed controlmay be advantageous in some cases, and is not outside the scope of theinvention). The more sophisticated the guidance program, the less needthere will be for commands additional to the waypoint coordinates.

The guidance computer accepts the waypoint coordinates from the radiouplink as input, and outputs signals directly to the vehicleservomechanisms which control the motor, brakes and steering of thevehicle V. (Motor, etc. are listed as examples. Any sort of physicalcontrols, depending on the particular type of vehicle, may be part ofthe invention. In this Description and in the following Claims, "vehicleservomechanism" means any means for physically controlling a vehicleaccording to signals from the guidance system.)

The physical motions of the vehicle V in response to the guidance systemsignals will not ordinarily be completely predictable. Wheel slip,steering gear backlash, or water or air currents (in the cases ofvehicles which are boats or planes), will all throw off the intendedpath. In other words, the planned ground path and the executed groundpath may differ. For this reason, the vehicle will ordinarily include anavigation system consisting of sensors and means for reporting thevehicle's attitude, speed, etc. to the vehicle guidance system and tothe operator's station computer.

The sensors may be odometers, magnetic compasses, accelerometers withposition integrators, speedometers, gyrocompasses, satellite positionsensors, or any other means of detecting the position, attitude, orstate of the vehicle V.

(The guidance system may act upon the input of non-navigational sensorsas well. Such sensors would detect vehicle environmental conditions ordangers, such as mines, quicksand, etc. Such a sensor 22 is shown inFIG. 5.)

The feedback of the navigation system to the guidance system is optionalin cases where the vehicle motion is predictable from the signals sentto the vehicle servomechanisms. However, the feedback of the vehicle Vposition to the station computer is not optional, because the vehicleposition at the time of the frame snapshot must be known to the stationcomputer's screen path generator if the screen path is to be correctlydisplayed on the screen 14. Therefore a device to maintain and reportthe vehicle's current position at any time is an essential element ofthe present invention. The navigation system with its sensors may beomitted if dead reckoning calculations in the guidance system are reliedupon to maintain a current vehicle position, and the guidance system hasmeans for reporting the position at the time of a snapshot to thestation.

The projected path of the vehicle V extends from the vehicle positionwhere the most recent frame was taken by the camera; this is thereference point for all calculations and motions. The path will extendup to the last waypoint 20 transmitted by the operator O. The guidancesystem need not stop or slow the vehicle V until it nears the lastwaypoint 20 in the path.

The vehicle, when it transmits a frame, will also transmit a report onits position at the time of the "snapshot". This frame position will bereferenced to the waypoints. The current vehicle position, as discussedabove, is maintained by the navigation system or guidance system.

Each time the screen is refreshed with a new frame, a new screen pathline 12 for that frame is calculated by the computer of the screen pathgenerator. Input for the calculation is the vehicle's reported frameposition and the stored waypoint positions; output is the placement ofthe line 12 on the screen. The line 12 is constructed from the waypointsby an algorithm which operates inversely to the algorithm which picksthe waypoints from the line.

The position of the vehicle V at the time the frame is taken willgenerally be intermediate between waypoints. The intermediate positionmay be referenced in various ways: the coordinate origin may be shiftedto the last-passed waypoint; the position may be specified as a pathdistance from the last waypoint, and the path reconstructed by a stationcomputer according to the same algorithm which the guidance systemcomputer uses to generate the ground path from the waypoints; or someother methods may be used.

It should be noted that the screen path is referenced to both the groundposition of the vehicle at some instant and the frame snapshot taken atthe same instant. Even if the executed ground path has drifted away fromthe planned ground path, and the navigation system has not corrected theerror, the screen path, the image of the frame, and the ground path areall kept synchronized by being reset to the same position andorientation with each new frame. There is no accumulated drift toinvalidate the operator's commands. Thus both direction and distancesensors may be simple, relatively low-accuracy types, as they areconstantly "recalibrated".

Because the operator O is using the cursor control to pick the vehicle'sfuture path, and not its present actions, there is no feedback lag tothrow the operator's reactions off.

Ordinarily, the TV camera 30 will be a visible light camera with alow-distortion lens 36. The operator's screen 14 will then present animage similar to that which he or she would see through a window on thevehicle. As discussed above, the coordinate transform of the screen pathto the planned ground path maps the line 12 into waypoints 20 on theground. The ground trapezoid is the transformed shape of the rectangularscreen 14. (A low-distortion lens maps a ground trapezoid onto itsrectangular film plane.)

If the camera is tilted to roughly horizontal, the horizon will be inthe picture, and the trapezoid will extend to infinity. The portion ofthe frame above the horizon will be outside the range of the mappingtransform.

Partial simulations, such as enhanced images resulting from calculationsbetween adjacent pixels, are within the scope of the present invention.Images which are transformed to overcome lens distortion or to present arotated, expanded, or intentionally distorted image to aid operatorcomprehension, are also within the scope of the present invention, asare added or enhanced portions of the screen image (for example, targetsights or flashing highlights) and inset or superimposed images todenote objects identified by auxiliary systems.

Images which are generated as time projections or predictions from datareceived, or which are interpolations of discrete screen views separatedin time, are not within the scope of the present invention.

In the present invention, the operator O is presented with a "snapshot"still frame of the ground terrain whenever the screen is refreshed. Thisis accomplished by video data storage in a two buffers.

The camera 30 will typically be a horizontal-sweep, top-to-bottom scanTV. The camera 30 will scan once in a short time, to "snap" a picture orframe, and send the image data to an image buffer in one of thevehicle's on-board computers for storage. The stored data represents onecomplete frame. This video data is sequentially retrieved from the imagebuffer and sent over the radio downlink to the station, where the videodata is again stored in a display buffer of the station computer memoryas it comes in. Once all the data for a complete frame has arrived, thestored frame is displayed on the station screen 14 continuously untilthe data for the next complete frame has arrived. The operator O thussees a still picture frame taken from the position of the vehicle V atan instant in the past, which remains on the screen until the next frameis ready for display. The time between the snapshot and display isnearly equal to the time required to transmit the video data over theradio downlink, because the downlink sends only the vehicle position inaddition to the video data, and the transmission time is very small. Iftransceiver radios 24 are used, the uplink time must be included in thecomplete frame cycle of radio transmission, but this also requires onlya brief interval compared to the time needed to transmit the greatamount of video data. This is the preferred method, because only oneradio band is needed for each vehicle; the total bandwidth for a set ofvehicles is halved.

(The system of the present invention is also adapted to wide-band bursttransmission, in which each vehicle uses the same radio frequency inturn. In this case the image can be sent as it is scanned, and nostorage buffer is needed on the vehicle. A display buffer is stillneeded at the station if the time between frames is greater than thepersistence time of the eye, that is, about a twentieth of a second.)

In some cases the camera 30 should have the ability to tilt, pan andzoom in response to operator commands. It may also be necessary to movethe camera 30 from place to place on the vehicle V, or extend it on aboom. If this capability exists, the parameters of the coordinatetransforms from the line 12 to the ground waypoints 20 will change asthe camera moves and changes its focal length. These changes must bereported to the station computer for generating the screen path and thewaypoints. The camera position will not automatically "reset" as willthe vehicle position. The camera must be kept in reference to thevehicle body for the present invention to work.

Camera motion relative to the vehicle V requires a camera mount 32,shown in FIG. 5. The mount 32 will have servomechanisms 34, may includesensors for detecting the position of the camera 30. In FIG. 5, thesensors are incorporated into the servos 34. FIG. 5 shows a mount havingtilt, pan, and transverse translation mechanisms. These are illustrativeonly: mounts allowing any sort of motion are within the scope of thepresent invention. FIG. 5 also shows a zoom servomechanism and sensor 34mounted on the lens 36.

A schematic of a camera control system is shown in FIG. 7. If the cameracontrols are positive, such as by step motors, the station computer cankeep track of the camera position by integrating all the camera commandsas time goes by, and calculating the transform parameters at the sametime; in this case the control will follow the solid arrow lines of FIG.7. If on the other hand the camera controls are of the type which candrift, such as belt drives, then the camera mount must include sensorsand means to report the attitude and focal length of the camera andlens. The extra information flow for this case is shown by the dashedlines in FIG. 7. The station computers cannot know the the dashed linesin FIG. 7. The station computers cannot know the position of the cameraby integrating past commands sent over the uplink, and so must receivereports from the camera sensors by way of the downlink.

The camera may be controlled by operator commands over the uplink. Thecamera may also be controlled automatically, that is, by a computer.Either the guidance system computer on board the vehicle, or a stationcomputer, may direct the camera motions. If the guidance system controlsthe camera, then feedback information on camera position which is neededat the station may be sent either from camera mount sensors, or else byrelaying the guidance system's camera commands, over the downlink to thestation. If camera control originates at the station, then feedback oncamera position from the vehicle may be needed.

The present invention is intended primarily for ground surfaces whichare two dimensional, but it also adaptable to piloting vehicles whichmove in three-dimensional space above a surface, for example, droneairplanes and shallow-water submarines. Vehicles like these are notconceptually different from ground vehicles in which the camera iselevated to various heights on a boom or telescoping mast. Suchspace-traversing vehicles must include an altitude sensor, whose outputis downlinked to the station computer to be used as a screen/groundtransform parameter.

Two-dimensional surfaces are rarely flat. If the navigation sensorsinclude inclinometers, and the guidance system of the vehicle has aprogram adapted to dealing with hills and ravines, the vehicle willguide itself more accurately. Such a program may account forforeshortening of the path as laid out on the screen, as for example, bymultiplying odometer readings by the cosine of the slope to obtain thehorizontal distance traveled when the waypoint generator has assumedlevel ground. A more sophisticated program might allow the operator O tobreak the path into discontinuous segments, for use when a portion ofthe ground is invisible on the screen, as at the crest of a hill.

As can be seen in FIG. 6, the present invention incorporates three basicfeedback loops whose names are centered inside the respective loops: theoperator loop wherein the operator O determines the path; the vehicleloop wherein the vehicle V is guided according to waypoints and commandstransmitted over the uplink; and the system loop which is incorporatesthe radio links. The vehicle loop is not an essential element of thepresent invention, as the navigation system is optional. The other twoloops are essential.

The discussion above outlines the preferred embodiment of the presentinvention, which allows the maximum vehicle speed possible given stillpicture frames. Video data storage in a storage buffer in the on-boardcomputer is necessary in this embodiment. There is another embodimentwhich is within the scope of the invention, when used with continuoustransmission on a narrow radio link (as opposed to burst transmission).

In this second embodiment the camera scans continuously and the data issent continuously to the operator's station. The display buffer and thestation computer are adapted to present a screen with two framesintermixed; the border between the two views scans down and a singleframe would appear only at the beginning (or end) of a sweep. Real-timeimages are just above the horizontal border line; time delay increaseswith distance upward from the border to the top of the screen, and thenincreases yet more as the view "wraps around" to the bottom: the imagejust below the border is that part farthest in the past.

This method of image transmission is the closest to realtime that ispossible with a slow radio link (one part of the image is always in realtime.) However, distortion will be introduced into the views by thismethod, since the vehicle is moving while the TV camera scans. As aresult, circular objects would appear egg-shaped. However, since scannedframes present the most up-to-date information possible, some distortionmight be tolerable; or, the display screen or the station computer couldcompensate for the distortion.

This second embodiment also requires adjustment of the vehicle computerguidance system's reporting of the vehicle position. Some particulartime must be designated for position references. Also, the programs usedto calculate the screen path/ground path transforms would need to bemodified.

In the present invention, the various components discussed above may bereplaced by functional equivalents which are within the scope of thepresent invention according to the definitions above. It is to beunderstood that the present invention is not limited to the soleembodiment described above, but encompasses any and all embodimentswithin the scope of the following claims.

I claim:
 1. A control system for an operator to control a vehicle from aremote station, said system comprising:a camera on said vehicle forgathering image data; means for sending said image data from said camerato said station, said means for sending said image data including aradio downlink from said vehicle to said station; a screen at saidstation for displaying images from said camera, for viewing by theoperator; means for generating a cursor and a screen path on saidscreen, for viewing by the operator; a cursor control for the operatorto move said cursor on said screen to determine placement of said screenpath on said screen; transform and waypoint generation means forgeometrically transforming said screen path into a planned ground path,for determining waypoints along said planned ground path, and forassigning waypoints coordinates to said waypoints; a radio uplink fromsaid station to said vehicle for sending said waypoint coordinates tosaid vehicle; a vehicle guidance system to guide said vehicle over anexecuted ground path, said executed ground path including saidwaypoints, said vehicle guidance system adapted to accept said waypointcoordinates, said vehicle guidance system including a guidance systemmemory for storing said waypoint coordinates, said vehicle guidancesystem including vehicle servomechanisms to physically control motionsof said vehicle; and means for reporting a frame position of saidvehicle to said transform and waypoint generation means over saiddownlink; whereby said screen will display an image of said vehicleenvironment taken by said camera at said frame position, the operatorwill trace out said screen path with said cursor control, and saidvehicle will following said corresponding ground path automatically. 2.The control system as in claim 1, wherein said means for reporting saidframe position of said vehicle includesa navigation system formaintaining a current vehicle position relative to said waypoints, saidnavigation system including sensors for detecting position, motion ororientation of said vehicle, or conditions of said vehicle environment.3. The vehicle as in claim 2 wherein said navigation system reports saidcurrent vehicle position to said guidance system, for vehicle feedbackto said guidance system.
 4. The control system as in claim 1,whereinsaid means for sending image data includes an image buffer forstoring said image data of one frame from said camera while saiddownlink transmits said data to said station, and said screen includes adisplay buffer for maintaining said one frame stationary on said screen,whereby the operator will continuously view a still picture taken bysaid camera in the past.
 5. The control system as in claim 1,whereinsaid control system includes command generation means for theoperator to generate commands, and wherein said uplink transmits saidcommands to said guidance system for controlling said vehicle.
 6. Thecontrol system as in claim 5, including a mount connecting said camerato said vehicle, said mount including camera motion means for movingsaid camera relative to said vehicle, said camera motion means includingmount servomechanisms;wherein said commands include camera motioncommands; wherein said uplink transmits said camera motion commands tosaid camera motion means; and wherein said transform and waypointgeneration means obtains information on camera position from saiddownlink for transforming said screen path into a planned ground pathaccording to said camera position; whereby said camera will pan, tilt,or translate at said commands from the operator and said control systemwill function properly.
 7. The control system as in claim 6, includingmeans for automatically controlling said camera position wherebysaidmeans for automatically controlling contains means for comparing saidwaypoints transmitted by said transform and waypoint generation meanswith said frame position of said vehicle.
 8. The control system as inclaim 6, wherein said camera includes a variable-focal-length zoom lensand zoom control servomechanism, said camera motion commands includezoom commands, and wherein said transform and waypoint generation meansobtains information on camera lens focal length from said zoom controlservomechanisms for transforming said screen path into a planned groundpath according to said focal length; wherebysaid camera lens will zoomat the command of the operator and said control system will functionproperly.
 9. The control system as in claim 8, including means forautomatically controlling said camera position wherebysaid means forautomatically controlling contains means for comparing said waypointstransmitted by said transform and waypoint generation means with saidframe position of said vehicle.
 10. The control system as in claim 1,wherein said transform and waypoint generation means includes means forautomatically determining said waypoints.